A Performance Study of a Large-scale Data Collection Problem

We consider the problem of moving a large amount of data from several source hosts to a destination host over a wide-area network, i.e., a large-scale data collection problem. This problem is important since improvements in data collection times are crucial to performance of many applications, such as wide-area uploads, high-performance computing, and data mining. Existing approaches to the large-scale data collection problem are (a) transferring data directly from the source hosts to the destination host, using IPprescribed routes (which we refer to as direct methods) or (b) using “best”-path type application-level re-routing techniques, which we refer to as noncoordinated methods. However, we believe that in the case of large-scale data collection applications, it is important to coordinate data transfers from multiple sources. More specifically, our coordinated method takes into consideration the transfer demands of all source hosts and then schedules all This work was supported in part by the NSF ITR CCR0113192 and the NSF Digital Government EIA0091474 grants. Dept. of Computer Science and Information Engineering, National Taiwan University. This work was partly done while the author was with the Department of Computer Science and UMIACS at the University of Maryland. Department of Computer Science and UMIACS, University of Maryland at College Park. TeleGIF, Marina del Rey, California. This work was partly done while the author was with the Department of Computer Science and UMIACS at the University of Maryland. Computer Science Department and IMSC and ISI, University of Southern California. This work was partly done while the author was with the Department of Computer Science and UMIACS at the University of Maryland. data transfers in parallel, using multiple paths existing between the source hosts and the destination host. All this is done at the application layer. In this paper, we present a performance and robustness study of the different data collection methods, namely the direct, the non-coordinated, and the coordinated methods. Our results show that coordinated methods can perform significantly better than non-coordinated and direct methods under various types of network congestion conditions. We also show that coordinated methods are more robust than non-coordinated methods under inaccuracies in network conditions information. Therefore, we believe that coordinated methods are a promising approach to large-scale data collection problems.